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Executive Summary 

The Office of Shared Accountability conducted an evaluation of the implementation of the third 
and final year (2012-2013) of the Science, Technology, and Engineering Leadership Program 
(STELP) in Montgomery County Public Schools (MCPS). The study was requested by the 
Office of Curriculum and Instructional Programs. Funding for STELP, including the evaluation 
study, is provided by a grant from the Howard Hughes Medical Institute (HHMI) to MCPS. A 
new grant cycle, including additional projects from HHMI, began in the fall of 2013. 

Background and Evaluation Questions 

This evaluation report addresses the third year of STELP. The focus of this evaluation was on 
implementation of the program in terms of continued training of a group of teacher leaders to 
further develop and refine online professional development products for other MCPS science, 
technology, and engineering (STE) educators to view. 

This study used a nonexperimental design utilizing a variety of data collection methods. Data 
collection methods included review of program documents and training records, surveys, and 
review of the online product training plans created by teacher leaders. Findings are organized by 
evaluation questions. 

Summary of Key Findings 

Question 1. What were the characteristics of the teacher leaders who participated in the year 
three training sessions of the Science, Technology, and Engineering Leadership Program? 

A total of 35 teachers participated in year three of STELP; 14 of those 35 (40%) teachers were 
returning participants from year two. The participants were mostly secondary science and 
technology teachers as well as K-5 elementary teachers. They represented 30 different schools 
across the county. 

Question 2. To what extent was the training of teacher leaders implemented as planned? 

All of the whole-group training sessions were held as planned: a new participant training in 
September 2012 plus five more trainings between October 2012 and May 2013 were conducted. 
A range of 28-35 of the 35 teacher leaders attended the sessions. 

The purpose of the training sessions was to provide teacher leader participants with a greater 
understanding of STE instruction and professional development as well as the technical skills 
needed to create online professional development products. Many of the sessions had dedicated 
time for working on products. 

Eight teams, comprised of three to five teachers, were expected to further develop and refine an 
online STE professional development product on a practice that was originally created in year 
two. Also, each team was assigned an STE staff member (STE specialist) as a support and “go 
to” resource person throughout the year. 
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Question 3. What was the impact of the STELP training sessions on teacher leaders? 

Teacher leader participant surveys were administered at the end of four of the five group training 
sessions. The response rate among the 35 STELP participants ranged from 74% to 97% for the 
four surveys; although most nonrespondents did not attend the corresponding training session. 
The surveys assessed teacher leaders’ perceptions of the training received in the program. 

Teacher Leaders’ Perceptions of Training Sessions and Expectations. Across two 
sessions (October and December), large percentages of teacher leader participants responded 
with positive perceptions of the training regarding clarity of goals, meeting objectives, 
knowledgeable and prepared trainers, a comfortable environment, opportunities to reflect, having 
questions answered, and helpful information and skills gained. All or most of the participants 
rated the various training activities as very helpful or somewhat helpful. The top important 
aspects mentioned across three of the sessions were learning about expectations and goals, 
networking, learning about science practices, and working on the product modules. Just about 
everyone at the four training sessions agreed or strongly agreed that the expectations for the 
products and next steps were clear. 

Teachers Leaders’ Perceptions of Knowledge and Skills Gained and Reflection. 

Overall, teachers’ perceptions of their skills with video style camera technology and Windows 
Movie Maker increased from the first session in October to the last session in May. Also, 
teachers’ perceptions of their level of understanding and articulating the proficiencies of science 
and engineering increased from the first session in October to the last session in May. 

At the last session in May, more than half (56%) of the 25 respondents said that they are 
including and focusing more on the Next Generation Science Standards (NGSS) practices in 
their instruction. Almost one third (8 of 25, 32%) responded that it has impacted their instruction 
by having them reflect on and improve their own teaching and lessons. Forty percent (10 of 25) 
responded that sharing the practices, standards, and information acquired is how the STELP 
professional development impacted their work with colleagues at their school and 9 of 25 (39%) 
said they have done so at team and department meetings. Almost half (11 of 24, 46%) of the 
respondents stated that the most important thing they gained from this STELP project was a 
better understanding of the practices and integrating them into teaching. 

Question 4. When and how were the online products made available to MCPS teachers? 

Development and Dissemination of Online Professional Development Products. In 

year two, eight teams of teacher leaders created eight online professional development products 
(also called modules) to help other MCPS educators learn about the NGSS practices. The online 
products were introduced to science resource teachers in September 2012. 

In year three, the online products were updated, refined and relaunched in July 2013 at a science 
resource teacher (RT) meeting, where RTs were encouraged to introduce the online products to 
their school staff in the fall. The online products also were discussed during an October 2013 
RT meeting. The eight online products were available to all MCPS teachers to access throughout 
year three and continue to be accessible. 
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Development of Online Professional Development Product Training Plans. Along 
with the online products, each year three leadership team produced a detailed training plan to 
coincide with their product. These training plans outlined a way for RTs or others in a training 
role, to incorporate the online products as a tool when training their teachers on an NGSS 
practice. A summary of the developed training plans may be found in Appendix E. 

Question 5. What was the level of usage of the online professional development products by 
classroom teachers? 

There was no evidence of interaction with the online products among classroom teachers based 
on the survey tool; however, we cannot fully and accurately answer this question because the 
software used for the online products was not capable of tracking participants. Additionally, the 
response rate for an online feedback survey, which was attached to the online product website, 
was zero. This may or may not indicate no or low usage of the products, since the survey was a 
separate link and completion was optional. 

A paper survey was administered to science RTs at a November 2013 professional development 
meeting. Fifty-four RT’s completed the survey representing over half (61%) of all secondary 
science RT’s with an 87% response rate of those who attended the meeting. 

About half of the secondary RTs reported that they had individually viewed one or more of the 
online professional development products outside of the RT meetings and over one third reported 
that they had shared one or more of the online products. More middle school RTs reported 
sharing the products than high school RTs. The most commonly viewed and shared product was 
the one for Practice 4: Analyzing and Interpreting Data followed by Practice 7: Engaging in 
Argument from Evidence and Practice 1: Asking Questions and Defining Problems. 

More than two thirds reported that they definitely or probably will complete (67%) or share 
(70%) a product module in the future. One half indicated that they definitely or probably will 
use the online product’s training plan. 

Question 6. What was the impact of the online professional development products on teachers 
who accessed them? 

Because there were no responses to an online feedback survey attached to the online product 
website, we asked secondary science RTs who attended a November 2013 RT professional 
development meeting how useful they thought the online products were. More than half of the 
34 responding (59%) indicated they thought the products were extremely or very useful for 
classroom teachers and another 38% thought they were somewhat useful. Furthermore, almost 
three fourths (73%) thought the modules were extremely or very useful for their role as resource 
teachers. 
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Recommendations 

• Continue to: a) promote and explore additional ways and formats to ensure the 
widespread and effective use of the online products; and b) explore additional 
ways in which the online products may be used by resource teachers. Include 
examples of how they can share with their staff and effectively use them in their 
staff meetings and professional development. 

• Elicit user feedback on the quality and usefulness of the online products so that 
modifications can be made to address concerns or problems. 

• Encourage user feedback on the ways that educators are using the online products 
so that best practices can be shared. 
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Evaluation of the 

Science, Technology, and Engineering Leadership Program, 

Year Three 

Natalie L. Wolanin and Julie H. Wade 


Background 

The overarching vision for science, technology, and engineering (STE) instruction in 
Montgomery County Public Schools (MCPS) is that all students achieve full literacy in these 
areas. Students who are literate in science, technology, engineering, and mathematics (STEM) 
are knowledgeable, informed citizens who are able to think critically about concepts and solve 
problems. MCPS supports this vision by engaging all students through seamlessly integrated 
instruction that is project/problem and standards based (MCPS, 2012). 

The Science, Technology, and Engineering Leadership Program (STELP) was introduced in 
MCPS in 2010. The program aims to grow instructional capacity in MCPS by training and 
supporting a cadre of teacher leaders to design and deliver online professional development. 
Building on the skills and knowledge developed through the Elementary Science Leadership 
Program, as well as tapping into the expertise of content specialists in secondary schools, STELP 
is preparing a group of teacher leaders to develop online materials to support inquiry-based 
instruction within effective, research-based teaching practices. With the creation of these 
resources, STELP aims to build a professional development network in science, technology, and 
engineering for wide use across MCPS. 

The ultimate goal for the three years of STELP is to improve STE instruction, and in turn, help 
students achieve STEM literacy (MCPS, 2010). This goal supports MCPS’s mission that every 
student will have the academic, creative problem solving, and social emotional skills to be 
successful in college and career (MCPS, 2013). Furthermore, it is in alignment with MCPS’s 
core value that “we will encourage and support critical thinking, problem solving, active 
questioning, and risk taking to continuously improve; stimulate discovery by engaging students 
in relevant and rigorous academic, social, and emotional learning experiences; and challenge 
ourselves to analyze and reflect upon evidence to improve our practices” (MCPS, 2013). 

Research reported in Taking Science to Schools and Ready, Set SCIENCE: Putting Research to 
Work in K-8 Science Classrooms (Michaels, Shouse, & Schweingruber, 2007; National Research 
Council, 2007) is the basis for the National Research Council’s Framework for K-12 Science 
Education released July 2012. This framework, in turn, is the springboard for the development 
of the Next Generation of Science Standards (NGSS) managed by the nonprofit organization, 
Achieve, Inc. These standards were released in spring 2013. The strands of scientific proficiency 
represent learning goals for students and address the knowledge and reasoning skills that 
students must acquire to be considered fully proficient in science. They are also a means to that 
end — they are practices that students need to participate in and become fluent with in order to 
develop proficiency. 
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During the first two years of STELP, the program trained and supported a group of teacher 
leaders by providing skills and knowledge to produce online professional development products 
that were based on a rubric of effective online professional development in science, technology, 
and engineering. The online products were developed during year 2 of STELP and launched in 
the fall of 2012. Year three of STELP included both new and returning teacher leaders and built 
upon the professional development products created in year two. 

The goals for the third year of STELP were: 

• Build a cadre of STE teacher leaders to support the MCPS STEM vision by 
designing and delivering online professional development. 

• Expand teacher instructional technology knowledge. 

• Expand teacher knowledge on student proficiency in science and engineering. 

• Continue to build collaboration between science and technology education. 

• Extend the STE leadership work done at the elementary and middle school levels 
to high school teachers. 

This evaluation report addresses the third and final year of the STELP project. Two previous 
reports were published on the outcomes for year 1 and year 2 of the project. The focus of this 
evaluation was on implementation of the program in terms of continued training of a group of 
teacher leaders to develop and refine the online professional development products for MCPS 
STE educators to view and use. 

The evaluation was requested by the Office of Curriculum and Instructional Programs (OCIP) 
and conducted by the Office of Shared Accountability (OSA). Funding was provided by a grant 
from the Howard Hughes Medical Institute to MCPS. 


Literature Review 

In a recent nationwide study, Wei, Darling-Hammond, and Adamson (2010) reported that 
teachers rated professional development in their subject area as their highest priority for further 
training. Consistent with this finding, teachers in an earlier study reported that professional 
development focusing on content knowledge was one of two elements that had the greatest effect 
on their knowledge and skills and led to changes in instructional practice (Garet, Porter, 
Desimone, Birman, & Yoon, 2001). 

In challenging budgetary times, it has become increasingly important to make the most efficient 
and effective use of limited resources in all areas of education, and professional development is 
no exception. Dahlberg and Philippot (2008) conducted a study to explore the perceived needs 
and perceptions of teachers regarding their professional development. The researchers 
concluded that there is no one-size-fits-all model to meet the professional development needs of 
teachers, arguing that professional development should be differentiated according to the varying 
needs and career stages of teachers. They advocate for a collaborative approach to determining 
professional development agendas, suggesting that, “Teachers, the ones who work most closely 
with the curricula and students, often know best where gaps in their own pedagogy and 
knowledge exist” (Dahlberg & Philippot, 2008, p. 22). 


Program Evaluation Unit 


2 


Evaluation of STELP, Year Three 



Montgomery County Public Schools 


Office of Shared Accountability 


As administrators have sought to stretch professional development dollars while providing 
teachers with accessible and meaningful professional development opportunities in their subject 
areas, interest in online professional development has grown (Dede, Ketelhut, Whitehouse, Breit, 
& McCloskey, 2009; National Research Council, 2007; Sawchuk, 2009). The flexibility of 
online professional development, as well as the capacity to tailor it to meet varying needs, makes 
it an attractive option in many school systems. As increasing numbers of teachers have 
participated in online professional development activities in recent years, evaluative research has 
not kept up with the growing use of these online models (Dede, et al., 2009). 

Dede and colleagues (2009) at the Harvard Graduate School of Education conducted a review of 
studies of online teacher professional development and noted that evidence of effectiveness was 
often lacking or anecdotal. In response to the scarcity of empirical findings, they developed a 
research agenda to help guide the study of online professional development toward a framework 
that would integrate theory and evidence-based practice. Among their recommendations are 
“research methodologies that do not simply replicate methods used in studying face-to-face 
professional development, but instead take advantage of the unique data collection possible in 
online programs” (Dede et al., 2009, p. 20). Their report also points out that since teachers apply 
what they learn over time, data should be collected over time as well. Consistent with the 
evaluation model constructed by Guskey (2000), Dede and his colleagues (2009) recognized the 
various levels of experience and learning to be addressed in an evaluation of professional 
development. They maintained that more and better measures implemented over time would 
help build understanding of what teachers learned in professional development, how they applied 
the new knowledge and skills to practice, and what changes resulted (Dede, et al., 2009). 
Consistent with the recommendations of Dede and colleagues (2009) in their “Research Agenda 
for Online Teacher Professional Development,” this evaluation includes data collected over time 
so that information about teachers’ use of the knowledge and skills gained from the professional 
development may be better understood. 

An evaluation of the first year of STELP was published in February 2012 (Wolanin & Wade, 

2012) . The report assessed the year one implementation of the project through multiple surveys 
of participants, interviews with program administrators, and document reviews. Participants were 
positive in their perceptions of the training and reports about skills and knowledge they had 
learned. Feedback from the participants was used to develop recommendations for year two, 
including clarifying the STELP vision and understanding of STELP strands, collecting regular 
updates on the products’ progress, and providing opportunities for teams to work on their 
products. 

An evaluation of the second year of STELP was published in January 2013 (Wolanin & Wade, 

2013) . The report assessed the year two implementation of the project through multiple surveys 
of participants, interviews with program administrators, and document reviews. Again, 
perceptions were positive regarding training, and the skills and knowledge learned. Feedback 
from the participants was used to develop recommendations for year three, including resolving 
technical issues and logistical difficulties of the online products, encouraging widespread use of 
the modules, collecting feedback on the modules, and monitoring the progress on continued 
training and product modifications, as well as on ways in which the teacher leaders are sharing 
their learned information. 
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Design and Scope of the Study 

The evaluation was designed using Guskey’s (2000) model for evaluating professional 
development. Four of Guskey’s sequential levels were addressed in the third year of the 
evaluation: participants’ reactions, participants’ learning, organization support and change, and 
participants’ use of new knowledge and skills. Table 1 outlines the levels of Guskey’s model 
along with the evaluation activities that were used to address each level. 

Table 1 


Evaluation Activities Using Guskey’s Model for Evaluating Professional Development 


Level of evaluation 

Instrument/activity 

Data collected 

1. Participants’ reactions 

Surveys of participants 
(administered after each 
training) 

Participants’ satisfaction and reactions to 
professional development 

2. Participants’ learning 

Surveys of participants 
(administered before and after 
training) 

Participants’ reported understanding of the 
proficiencies as described by the NGGS 
standards and their relationships to the 
MCPS STEM vision. The knowledge 
required to plan and create online 
professional development resources. 

3. Organization support 
and change 

Surveys of participants 
(administered after each 
training) 

Organizational support and teacher leader 
needs in the project 

4. Participants’ use of new 
knowledge and skills 

Surveys of participants 
(administered after each 
training) 

Participants’ reported use of new knowledge 
as they created professional development 
products 


The year three evaluation has two objectives: 1) assessment of the implementation and impact of 
the training of teacher leaders as they continue to develop and refine online professional 
development products for other MCPS educators; and 2) assessment of the impact of the 
professional development products on the classroom teachers who access them online and 
resource teachers who have them in professional development. Toward this end, the evaluation 
will address the following questions. 

1. What were the characteristics of the teacher leaders who participated in the year three 
training sessions of the Science, Technology, and Engineering Leadership Program? 

2. To what extent was the training of teacher leaders implemented as planned? 

3. What was the impact of the training sessions on teacher leaders? 

4. When and how were the online products made available to MCPS teachers? 

5. What was the level of usage with the online professional development products by 
classroom teachers? 

6. What was the impact of the online professional development products on teachers who 
accessed them? 


Program Evaluation Unit 


4 


Evaluation of STELP, Year Three 





Montgomery County Public Schools 


Office of Shared Accountability 


Methodology 

Participation in STELP was comprised of a group of teacher leaders selected by program staff, so 
a nonexperimental design utilizing a variety of data collection methods was applied. Data 
collection methods included reviewing program documents and training records, surveying 
teacher leaders and resource teachers, and reviewing the online product training plans created by 
teacher leaders. 

Study Sample 

In the third year of the evaluation, all teacher leaders enrolled in STELP comprised the study 
sample. A total of 35 school-based staff members participated during year three. Thirteen were 
from elementary schools, nine from middle schools, twelve from high schools, and one from a 
special education school in MCPS. Participants consisted of elementary, science, technology, 
special education, and staff development teachers. 

Data Collection Activities 
Program Documents 

Reviews of program documents and professional development session records and materials, 
including session agendas, session handouts, and session attendance records were made by OSA 
evaluators. 

Review of Online Professional Product Training Plans 

During year three, teacher leader teams created training plans, which accompanied each of the 
eight online professional development products they created. OSA evaluators reviewed these 
training plans and created a summary table of their contents (Appendix E). 

Surveys 

Based on program goals and objectives and professional development materials and curricula, 
evaluation survey instruments were developed by OSA evaluators, in collaboration with staff 
from OCIP. The following instruments were developed during the third year of the evaluation: 

Surveys of Teacher Leader Participants. The teacher leader participant surveys were 
administered at the end of four of the five group training sessions (Appendix A). The paper and 
pencil surveys assessed teacher leaders’ perceptions of the training received in the program. So 
as not to overburden participants, a survey was not conducted at the April training session since 
there was another training survey scheduled for the May session. 

Surveys of Online Product Users. Online surveys were developed for each of the eight 
online professional development products; the surveys were designed for users to give feedback 
after viewing a product. A sample of a survey may be found in Appendix B. 

Surveys of Secondary Science Resource Tachers. Resource teacher (RT) surveys were 
administered in November 2013 at the end of a RT professional development session. The paper 
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and pencil surveys assessed RTs’ usage and expected usage of the online professional 
development products. The RTs were shown the products in July 2013 at a professional 
development session. The administration of the survey in November was to allow ample time 
for the RTs to use the products or plan how they might use them in the future for their own 
school staff development (Appendix C). 

Summary of Data Analysis Procedures 

Procedures included a descriptive statistical analysis of: 

• Teacher leaders’ survey responses 

• Characteristics of participants 

• Attendance at professional development sessions 

• Characteristics of training plans developed for the online professional 
development products 

• Resource teachers usage and plans for the online products and their corresponding 
training plans 

• Teacher feedback about training sessions and STELP program 

Strengths and Limitations of the Study 

There was no participation in the online product surveys, which were meant to capture user 
feedback about the products. This limited the ability to answer Question 3: What was the impact 
of the online professional development products on teachers who accessed them? In addition, it 
was not possible to track anyone who viewed the online products. Therefore, we were not able 
to fully answer the evaluation Question 5: What was the level of usage with the online 
professional development products by classroom teachers? Finally, although we were able to 
capture some feedback from secondary RTs about their usage of the online products (see strength 
below), we were not able to capture similar feedback from elementary teachers. 

A strength of the study was that the evaluators were able to administer four surveys at the end of 
four of the five training sessions. The surveys were completed by almost all of the training 
participants. An additional strength of the study was that evaluators were able to collect 
feedback from secondary RTs about their usage of the online professional development products. 
Almost all RTs that attended the meeting where the survey was administered completed the 
survey (87% response rate), which is 61% of all secondary RTs. This was especially important 
since it was our only means of gathering information on the products from users. 
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Findings 

Question 1 

What were the characteristics of the teacher leaders who participated in the year three training 
sessions of the Science, Technology, and Engineering Leadership Program? 

Invitation and Enrollment of Participants 

The cadre of teacher leaders who participated in year two were invited to return in year three. In 
addition, new teachers were recruited to expand the number of technology education and middle 
and high school teachers, and to replace teachers who did not return. New teacher leaders 
attended an introductory training session in September, prior to the regular schedule of training 
sessions. A total of 37 teachers started in year three, but in the end, a total of 35 teachers 
participated in year three of STELP; 14 of those 35 (40%) teachers were returning participants 
from year two. 

Characteristics of the Participating Teacher Leaders 

The participants represented a variety of positions in both elementary and secondary schools. As 
shown in Table 2, of the 35 who participated, 13 were elementary school teachers (however, one 
was from a special education school and one was from a K-6 school.) Twenty-two were 
secondary school teachers (10 middle and 12 high schools). Characteristics of the STELP 
participants are summarized in Table 2. 


Table 2 


Science, Technology, and Engineering Leadership Program: 
Characteristics of Participating School Staff 



Total 

Teacher Leaders 
(N = 35) 

Elementary School 
Teacher Leaders 
(N= 13) a 

Secondary School 
Teacher Leaders 
(N = 22) b 

n 

% 

n 

% 

n 

% 

Current position 

K-2 teacher 

6 

17.1 

6 

46.2 



3-5 teacher 

4 

11.4 

4 

30.8 



Special education teacher 

1 

2.9 

1 

7.7 



Staff development teacher 

1 

2.9 

1 

7.7 



MS science teacher 

9 

25.7 

1 

7.7 

8 

36.4 

HS science teacher 

9 

25.7 



9 

40.9 

MS technology teacher 

2 

5.7 



2 

9.1 

HS technology teacher 

3 

8.6 



3 

13.6 

Total years in current position 

1-4 years 

10 

28.6 

3 

23.1 

7 

31.8 

5-10 years 

14 

40.0 

6 

46.2 

8 

36.4 

11+ years 

9 

25.7 

3 

23.1 

6 

27.3 

No response 

2 

5.7 

1 

7.7 

1 

4.5 

Degree or certification in science 

s, technology, or engineering 





No 

12 

34.3 

12 

92.3 


Yes 

23 

65.7 

1 

7.7 

22 

100.0 


J A teacher from a Special Education school is included. Also, a middle school (MS) science teacher from a K-6 
elementary school is included. 

b Of the 22 secondary teachers, 10 were from middle school and 12 were from high school (HS). 
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One half (18 of 35, 51%) were middle or high school science teachers (one from a K-6 school); 
5 of 35 (14%) were middle or high school technology teachers, 6 (17%) taught kindergarten 
through Grade 2, and 4(11%) taught Grades 3-5. Two thirds (23 of 35, 66%) had more than five 
years teaching experience, and two thirds reported having a degree or certification in science, 
technology or engineering, all of them from secondary schools (Table 3). 

Seven of the 13 (54%) elementary school participants were also school team leaders and three 
(23%) reported being a science or STEM lead at their school. Six of the 22 (27%) secondary 
participants were also science RTs at their school (Table 3). 


Table 3 

Science, Technology, and Engineering Leadership Program: Previous Training and 
Leadership Experience of Participating Elementary and Middle School Staff 

Total Elementary Secondary 

Teacher Leaders Teacher Leaders Teacher Leaders 


(N = 35) 

cn 

II 

5 

(A =22) 

n % 

n % 

n % 


Team leader 

7 

20.0 

7 

53.8 

0 

0.0 

Science lead/STEM lead or 
Coordinator/resource teacher 3 

9 

25.7 

3 

23.1 

6 

27.3 


“Three middle school and one high school science teachers were resource teachers; two high school technology 
teachers were resource teachers. 


Characteristics of Schools with STELP Teachers 

Teachers who participated in STELP were from 30 different schools: 11 elementary schools, 10 
middle schools, and 8 high schools, as well as one special education school. Characteristics of 
the schools represented are shown in Table 4. 


Table 4 


Science, Technology, and Engineering Leadership Program: 
Characteristics of Elementary and Middle School Participants 




Elementary 

STELP 

Middle 

STELP 

High 

STELP 

School-level characteristics 

(11 schools) 

MCPS 

(10 schools) 

MCPS 

(8 schools) 

MCPS 

Number of 

Mean (SD) 

564 (192) 

546 (147) 

855 (220) 

822 (215) 

1747 (290) 

1739 (446) 

students 

Range 

318-1009 

160-1009 

551-1370 

351-1370 

1306-2244 

516-2806 

% of students 
eligible for 

Mean (SD) 

25.9 (21.2) 

37.1 (26.2) 

22.2 (13.4) 

33.3 (19.1) 

32.6(15.0) 

28.3 (16.1) 

FARMS 

Range 

1.8-75.6 

1.1-94.8 

1.4-43.7 

1.4-64.5 

7.8-52.0 

2. 6-7. 9 

% of students 

Mean (SD) 

20.9 (14.1) 

24.1 (16.1) 

6.3 (3.8) 

8.6 (5.4) 

6.1 (2.5) 

5.9 (4.2) 

enrolled in ESOL 

Range 

5.0-50.6 

3.1-76.4 

0.6-14.3 

1.0-21.2 

1.6-8. 5 

0.1-18.2 


Note. Based on 2013 MCPS data. STELP includes schools represented (not number of participants). The special education school is not 
included in the STELP column, and a first year charter school is not included in the MCPS column. SD = Standard Deviation. 


On average, as a group, schools with teachers participating in STELP had proportions similar to 
MCPS averages in terms of students receiving English for Speakers of Other Languages (ESOL) 
services (21% of STELP elementary schools and 24% for MCPS; 6% for STELP middle schools 
and 9% for MCPS; and 6% for STELP and MCPS high schools). However, STELP schools at 
the elementary and middle school level had a lower proportion of Free and Reduced-price Meals 
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System (FARMS) service recipients than all of MCPS (26% compared to 37% for elementary 
and 22% compared to 33% for middle schools). Just as there is a wide range of proportions of 
FARMS recipients in MCPS schools, there is a wide range among schools with teachers 
participating in STELP (2-76% for elementary schools; 1-44% for middle schools; and 8-52% 
for high schools). Additionally, the range of ESOL recipients with participating STELP teachers 
is 5-51% in elementary schools, 1-14% in middle schools and 2-9% in high schools. 


Question 2 

To what extent was the training of teacher leaders implemented as planned? 

Training Schedule and Attendance 

Participation. The program plan specified that a new participant training would be held 
in September 2012. Additionally, whole-group training sessions would be held in October 2012, 
December 2012, February 2013, April 2013, and May 2013. All of these sessions were held as 
planned; attendance is shown in Table 5. 

Overall, attendance was fairly good across the five sessions. A range of 28-35 of the 35 teacher 
leaders attended the sessions from October 2012 through May 2013. More than half (20 of 35, 
57%), of the participants did not miss any of the sessions and almost a third (11 of 35, 31%) only 
missed one session (Table 5). The lowest attendance was observed at the February 2013 and 
May 2013 sessions with 28 of 35 (80%) participants attending these two sessions. 

Table 5 

Science, Technology, and Engineering Leadership Program: 

Training Participation 

Participants 



(N = 

35) 

Participation 

n 

% 

Absences per Participants 

Missed 0 sessions 

20 

57.1 

Absent 1 session 

11 

31.4 

Absent 2 or 3 sessions 

4 

11.4 

Attendance per Session 

October session 

35 

100.0 

December session 

32 

91.4 

February session 

28 

80.0 

April session 

30 

85.7 

May session 

28 

80.0 


Training Session Outcomes and Agendas 

The purpose of the training sessions was to provide teacher leader participants with a greater 
understanding of STE instruction and professional development as well as the technical skills 
needed to create online professional development products. 

Many of the sessions had dedicated time for groups to work on their products. In all of the 
professional development sessions, teacher leaders reviewed the vision for STELP, determined a 
timeline of action steps to be completed before the next professional development day and 
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completed an evaluation survey at the end of the day in all but one session. A summary of the 
sessions follows: 

• September 2012 (New Participant Session): Participant leaders learned about the vision 
for STE in MCPS, identified proficiencies in science and engineering, learned about the 
goals of STELP, and learned technology skills needed to create the instructional 
resources for STELP, such as the use of video recording during classroom instruction and 
the use of Windows Movie Maker and Microsoft PowerPoint. 

• October 2012: At the first session, teacher leaders were assigned their professional 
development team for the year, as well as their assigned STE content specialist, and they 
built relationships with their team members through team building activities. Teacher 
leaders also learned about the MCPS STE vision, the history and charge of STELP, as 
well as increased their understanding of the science and engineering proficiencies and 
how to utilize technological tools in the STELP program. 

• December 2012: Teacher leaders revisited the expectations for the STELP products and 
viewed a sample of the proposed plan of delivery for the completed products and how the 
products would reach the MCPS professional community. Participants also worked with 
their teams to identify any parts of their practice that were not yet represented in the 
product and worked on developing a suggested training plan for professionals to use at a 
school-level team meeting. 

• February 2013: Teacher leaders participated in a carousel with other team members to 
add components or elements of each practice that they felt were critical; they continued to 
work on a suggested training plan to deliver the professional development product and 
they worked on refining their product module by completing the slide show and editing 
videos associated with it. 

• April 2013: Teams met with their STE specialists to revisit the training plans and any 
upgrades made to their products. Teams also worked on refining and editing their 
product modules, such as reviewing the flow of the module, checking hyperlinks, and 
scripting the module. Participants also reviewed and gave feedback to another team’s 
training plan and module. 

• May 2013: The final scheduled session was dedicated time for teams to work on refining 
and editing their professional development product as well as to meet with their STE 
content specialist to review the training and product module utilizing a STELP checklist. 
They also determined final steps to publish the module on Articulate. Various dates were 
provided in May and early June for individuals to come to the MCPS Carver Educational 
Service Center (CESC) facility to do voice overs for the narration of their product and to 
publish their product. 

Formation of Teams and Selection of Topics 

Eight teams, comprised of three to five teachers, were formed at the first training session in 
October based on participants’ requests. The teams were each assigned one of the science and 
engineering framework practices, listed in Table 6, and were expected to develop and refine an 
online STE professional development product about that practice. They used the online products 
developed in the prior year to refine. Each practice corresponds to one of four learning strands 
developed by the National Research Council (NRC) Committee on Science Learning. Some 
repeat participants worked on the same practice as the previous year and some worked on a new 
practice. Also, each team was assigned an STE staff member as a support and “go to” resource 
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person throughout the year. Each team self-assigned roles for their members such as: 
coordinator, check for understanding developer, quality controller, PowerPoint expert, Movie 
Maker expert, and any other roles they deemed useful. 


Table 6 

STELP Online Professional Development Design Teams and their Assigned Topics 


Science & Engineering 
Framework Practice # 

Framework Practice 
Name 

Target Proficiency 
Strand 

Practice 1 

Asking Questions and 
Defining Problems 

Strand 1 : Know, use, and interpret scientific 
explanations of the natural and design world 

Practice 2 

Developing and Using 
Models 

Strand 1 : Know, use, and interpret scientific 
explanations of the natural and design world 

Practice 3 

Planning and Carrying 
Out Investigations 

Strand 2: Generating and evaluating scientific 
evidence or technological solutions 

Practice 4 

Analyzing and 
Interpreting Data 

Strand 2: Generating and evaluating scientific 
evidence or technological solutions 

Practice 5 

Using Mathematics and 
Computational Thinking 

Strand 2: Generating and evaluating scientific 
evidence or technological solutions 

Practice 6 

Constructing 
Explanations and 
Designing Solutions 

Strand 1 : Know, use, and interpret scientific 
explanations of the natural and design world 

Practice 7 

Engaging in Argument 
from Evidence 

Strand 4: Participating productively in practices 
and discourse of science and engineering 

Practice 8 

Obtaining, Evaluating, 

Communicating 

Information 

Strand 4: Participating productively in practices 
and discourse of science and engineering 


Question 3 

What was the impact of the STELP training sessions on teacher leaders? 

Teacher leader participant surveys were administered at the end of four of the five group training 
sessions (Appendix A). The paper and pencil surveys assessed teacher leaders’ perceptions of the 
training received in the program. So as not to overburden participants, a survey was not 
conducted in April since there was another training survey scheduled at the May session. Of the 
35 STELP participants, those who completed surveys were: 34 respondents in October (97%), 27 
in December and February (77%) and 26 in May (74%). Most of those who did not complete a 
survey also did not attend that session. The session feedback surveys were summarized by the 
evaluators and provided to program administrators after each training session. Findings helped 
the program staff improve and modify trainings accordingly. 

Teacher Leaders ’ Perceptions of Training Sessions 

Teacher leaders were asked to indicate their level of agreement with a series of general questions 
about the training session. These sets of questions (or a subset) were asked at the October and 
December session. Teacher leaders’ perceptions of the trainings are summarized in Table 7. 

Across the two sessions, large percentages of teacher leader participants responded with positive 
perceptions about the training, with nearly 90% of the participants agreeing with all of the 
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statements in October and 100% of the participants agreeing with all of the statements in 
December. The statements referred to aspects of the training, such as: providing clear goals, 
meeting objectives, providing knowledgeable and prepared trainers, a comfortable environment, 
opportunities to reflect, questions answered, and helpful information and skills gained. 


When broken out further, two thirds or more strongly agreed that there was a comfortable 
environment in both the October (26 of 34, 77%) and December (19 of 27, 70%) trainings, 
reflection opportunities were given in December (17 of 26, 65%), and critiquing the existing PD 
modules in October were helpful for understanding expectations (23 of 34, 68%). 


Table 7 

Science, Technology, and Engineering Leadership Prog ram: 

Teacher Leaders’ Perceptions of Training Provided in October and December Sessions 


October PD Session 


Strongly 

Agree 


Agree 



N 

n 

% 

n 

% 

The goals of today’s training were clear. 

34 

ii 

32.4 

19 

55.9 

The objectives of today’s training were met. 

33 

14 

42.4 

17 

51.5 

The trainers were knowledgeable and well prepared. 

34 

14 

41.2 

16 

47.1 

An environment was created in which I felt 

34 

26 

76.5 

7 

20.6 

comfortable taking risks. 

/ 

Opportunities were provided for me to process and 
reflect upon the application of the knowledge and 
skills learned. 

34 

13 

38.2 

18 

52.9 

My questions during the training today were 
answered adequately. 

34 

20 

58.8 

13 

38.2 

As a result of today’s training, I have gained 
information and skills that will help me in this role 
as an STE leader. 

34 

13 

38.2 

18 

52.9 

Critiquing existing PD modules was helpful in 
understanding the expectations for creating online 
PD. 

34 

23 

67.6 

8 

23.5 

Working in practice groups was helpful in building 
my knowledge of a specific practice and preparing 
me to develop a training for other MCPS teachers 
related to the practice. 

34 

21 

61.8 

11 

32.4 

The training session today helped me build my 
understanding of the proficiencies at different grade 
levels from the ones I teach. 

34 

9 

26.5 

20 

58.8 

The training today helped me build my 
understanding of the proficiencies of Science and 

30 

10 

33.3 

18 

60.0 


December PD Session 


Strongly 

Agree 


N 


% 


Agree 


% 


27 16 59.3 11 40.7 

27 13 48.1 14 51.9 

27 17 63.0 10 37.0 


27 19 70.4 


29.6 


26 17 65.4 9 34.6 


27 17 63.0 10 37.0 


27 13 48.1 14 51.9 


Note. PD = professional development. Only strongly agree and agree are shown from the 4-point scale, therefore totals may not 
add up to N. N reflects the number of participants who answered the question. The last four items were only asked at the October 
training session. 

d N = 30; 4 participants responded “not applicable” at the October training session because they reported they already had a 
thorough understanding. 


Participants were asked to rate the helpfulness of various activities performed in each of the 
trainings. All the activities were rated very helpful or somewhat helpful by more than 90% of the 
participants, with the exception of the carousel activity held in February, where 78% found it 
very or somewhat helpful (Table 8). When broken out further, certain activities were especially 
helpful with a large majority giving them the top rating of very helpful: the Instructional 
Specialist at the May training (19 of 26, 73%), the timeline in February (19 of 27, 70%), the 
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training plan in both February (18 of 27, 67%) and May (16 of 26, 62%), and the STELP 
checklist in May (17 of 26, 65%). 


Table 8 


Science, Technology, and Engineering Leadership Program: 

Teacher Leaders’ Perceptions of Helpful Aspects of Training Provided in Three Sessions 


December PD Session 

February PD Session 

May PD Session 

Very Somewhat 

Helpful Helpful 

Very Somewhat 

Helpful Helpful 

Very Somewhat 

Helpful Helpful 

How helpful was.... N n % n % 

N n % n % 

N n % n % 

Capture sheet 22 2 9.1 19 86.4 



Proposed plan of delivery 23 7 30.4 15 65.2 



Training plan 23 6 26.1 16 69.6 

27 18 66.7 9 33.3 

26 16 61.5 10 38.5 

Carousel activity 

27 7 25.9 14 51.9 


Timeline 

27 19 70.4 8 29.6 


Instructional specialist 

27 13 48.1 12 44.4 

26 19 73.1 6 23.1 

STELP checklist 


26 17 65.4 8 30.8 


Note. PD = professional developmentOnly “very helpful” and “somewhat helpful” are shown from the 4-point scale, therefore totals 
may not add up to N. N reflects the number of participants who answered the question. Blank cells indicate the item was not 
administered at that particular training session. 


Teacher Leaders Responses to Open-ended Questions 

At the end of the October, December, and February training sessions, teacher leader participants 
were asked a variety of open-ended questions on the survey about the training session. At the 
last session in May, the open-ended questions were reflective questions about the STELP project, 
which will be addressed in the next section of the evaluation. Teacher leaders’ responses to the 
open-ended survey questions are summarized below. 

Important Aspects of Training. At the end of three of the training sessions, teacher 
leader participants were asked, in an open-ended question, to identify the most important thing 
gained from the trainings. The top mentions for each training session are shown below. Percents 
are calculated from the number of participants who answered the question and multiple 
responses were allowed. 

Most important aspect of professional development training provided in October ( N = 36): 

• Learning about expectations, goals, and outcomes (n = 16, 44%) 

• Meeting their team and networking ( n = 10, 28%) 

• Learning about science practices and standards (n = 9, 25%) 

• Viewing and critiquing the existing modules ( n = 8, 22%) 

• Thinking of ways to improve product and incorporate STEM topics ( n = 5, 14%) 

Most important aspect of professional development training provided in December ( N = 26): 

• Time to work on product; time to work with team (n = 8, 31%) 

• Clarity on project and plans (n = 8, 31%) 

• Understanding of practice(s) (n = 4, 15%) 

• Gaining video and Powerpoint ideas ( n = 4, 15%) 

• Gaining technical knowledge ( n = 3, 12%) 
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Most important aspect of professional development training provided in February (N = 26): 

• Clarity on timeline and expectations ( n = 9, 35%) 

• Time to work on product (n = 7, 27%) 

• Receiving training plan (n = 4, 15%) 

• Learning how information will be presented to other teachers (n = 3, 12%) 

Suggestions for Improving the Training and Additional Supports to Fulfill Teacher 
Leader Role. At the end of three of the training sessions, teacher leader participants were asked 
“Was there was anything that would have been more effective had it been done differently?” and 
“Are there any additional resources or supports that you think you will need to fulfill the teacher 
leader role as you develop your professional development product?”. Few participants answered 
these open ended questions and even fewer had similar feedback. The top mentions for each 
training session are shown below. The findings from the December training are not shown 
because only a few participants answered the questions and they did not have similar answers. 

October Professional Development Training Session. Suggestions ( N = 15) 

• Better organization: downtime, logistics, communicating objectives/agenda (n = 5, 33%) 

• More background needed on modules ( n = 3, 20%) 

• Clearer goals and vision needed (n = 3, 20%) 

• More time with groups needed (n = 3, 20%) 

Additional Supports Needed (N = 15) 

• Technology support and knowledge (n = 6, 40%) 

• Help with videotaping - extra person or time (n = 3, 20%) 

• More meeting time in = 2, 13%) 

February Professional Development Training Session. Suggestions ( N = 8) 

• More information to help create training plan (n = 2, 25%) 

• More time for editing ( n = 2, 25%) 

Additional Supports from February training (N = 9) 

• Technology support ( n = 3, 33%) 

• Need additional time (n = 3, 33%) 

Other varying comments given throughout the surveys included questions about directions, 
expectations, and technical or detailed logistics. There were also several praises given for the 
project such as clarity and enjoyment working with team and on the project. 

Expectations and Progress. Just about everyone at the four training sessions agreed or 
strongly agreed that the expectations for the products were clear and next steps were clear 
(Table 9). When broken down further, a higher percentage of participants in the later sessions 
strongly agreed that the expectations for the products were clearer in February (17 of 27, 63%) 
and May (18 of 26, 69%) compared to the earlier sessions in October (10 of 33, 30%) and 
December (10 of 26, 39%), and expectations for next steps were clear (63% and 73% compared 
to 39% and 44%). Almost all participants (96%) agreed or strongly agreed, across the three 
sessions asked, that expectations for the delivery of the product were clear. 
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The majority of participants or all participants agreed or strongly agreed that they felt good 
about the progress their team had made so far and the direction their product was headed in or 
about their product so far (93-100%). 

All participants agreed or strongly agreed in December and February that expectations for what 
the team was to accomplish was clear and a large majority (92%) also did so in May; however, a 
much larger percentage (73%) gave the top rating of strongly agreed in the May session. The 
majority of participants or all participants agreed or strongly agreed (96% or 100%) that the 
February and May sessions helped them with their teams’ progress and that the proposed 
deadlines and dates were realistic. 


Table 9 


Science, Technology, and Engineering Leadership Program: 
Teacher Leaders’ Perceptions of Training Provided in Four Sessions 



Note. PD = professional development. Only strongly agree and agree are shown from the 4-point scale, therefore totals may not add up 
to N. N reflects the number of participants who answered the question. Blank cells indicate the item was not administered at that 
particular training session. 
d N = 26 in this December item. 
b N = 25 in this May item. 
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Responses of teacher leader participants about their perception of their team’s progress are 
reported in Table 10. In February, the majority (20 of 27, 74%) of respondents felt they were 
right on schedule and about one fourth (7 of 27, 26%) felt they were behind schedule. In the last 
training session in May, 77% of the 26 participants felt they were ahead or right on schedule with 
about one fourth (6 of 26, 23%) feeling they were behind. 

Table 10 

Science, Technology, and Engineering Leadership Program: 


Teacher Leaders’ Perceptions of Training Provided in Two Sessions 


How would you describe your team’s progress on 
your PD product? 

February 
(A =27) 

May 
(A =26) 

Ahead of our planned timeline 

0 

0.0 

4 

15.4 

Right on schedule 

20 

74.1 

16 

61.5 

Running behind our planned timeline 

7 

25.9 

6 

23.1 


Teachers Leaders’ Perceptions of Knowledge and Skills Gained 

Technology Skills. In surveys administered at the first and last training session, teacher 
leader participants were asked to indicate their level of specific technology skills. Table 11 
shows that overall, teachers perceptions of their various technology skills increased from the first 
session in October to the last session in May. More than two thirds (23 of 33, 70%) rated 
themselves proficient or advanced with video style camera technology, and more than one third 
(15 of 34, 44%) with Windows Movie Maker at the beginning of the training sessions. By the 
last training session, almost all (25 of 26, 96%) rated themselves advanced or proficient in video 
style camera technology, and more than three fourths (20 of 26, 77%) with Windows Movie 
Maker. For Microsoft PowerPoint, there was little change overall in the ratings with 85% rating 
themselves advanced or proficient in October (29 of 34) and in May (22 of 26). 

Table 12 shows the individual change in ratings from the first session to the last; only responses 
from those participants’ who provided ratings for both sessions (N = 26) are shown and 
individual rating changes were calculated. Almost one third (8 of 25, 32%) increased their skill 
level rating for use of video style camera technology, and 1 1 of 26 (42%) increased their skill for 
Movie Maker. Only 4 of 26 (15%) increased their rating for using PowerPoint. There were a few 
participants in each of the skills who actually decreased their skill level rating. 

Understanding of Science Proficiencies. In surveys administered at the first and last 
training sessions, teacher leader participants were asked to indicate their level of understanding 
and ability to articulate the proficiencies and their relationship to the MCPS STEM vision and 
the proficiencies of science and engineering as described by the NGSS. Table 11 shows that 
overall, their understanding increased from the first session in October to the last session in May. 
In October, 6 of 34 participants (18%) said they had a beginner understanding of the 
proficiencies and their relationship with the MCPS STEM vision; by May only 1 of 26 
participants (4%) reported having a basic/beginner understanding. Likewise, over one third 
(12 of 33) said they had a beginner understanding of the proficiencies of science and engineering 
in October; this decreased to only 3 of 26 participants by May (12%). 

Table 12 shows the individual change in ratings from the first session to the last; only responses 
from those participants’ who rated their skill in both sessions are shown (N = 26) and individual 
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rating changes were calculated. Well over one third (10 of 26, 39%) and almost half (12 of 25, 
48%) of participants increased their skill level rating for both of these proficiencies. Once again, 
there were a few participants in both skills who decreased in their skill level rating. 

Table 1 1 


Science, Technology, and Engineering Leadership Program: 



Note. PD = professional development. 
d N = 33 in October. 


Table 12 

Science, Technology, and Engineering Leadership Program: 



“N = 25. 
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Science Integration and Collaboration in Teacher Leaders ’ Schools 

Elementary School. STELP participants from elementary schools were asked to rate the 
extent of science integration into their school schedule at the first session in October and again at 
the last session in May. It should be noted that only 13 elementary participants answered the 
question in October and 9 answered in May (Table 13). At both times, the majority reported 
ratings of somewhat or a little (9 out of 13, 69% and 5 out of 9, 56% respectively). 

Table 14 shows the individual change in ratings from the first session to the last; only the seven 
participants’ who rated their skill in both sessions are shown. Of these seven, six did not change 
their rating, and one increased their rating when their May response was compared to their 
October response. 

Secondary School. STELP participants from secondary schools were asked at the first 
session in October to rate the extent of collaboration between the mathematics and science 
teachers in their school and also between their mathematics, science, and technology teachers; 
they were asked again at the last session in May. The perception level of collaboration between 
science and mathematics teachers varied. One half of respondents (55%) indicated that teachers 
at their school collaborated somewhat/a little, while a similar number reported the teachers did 
not collaborate at all (46%) (Table 13). When responses of the 16 participants who answered in 
both October and May were examined, about half (9 of 16, 56% and 8 of 16, 50% respectively) 
did not change their rating and 5 of the 16 (31%) increased their rating (Table 14). 

Table 13 


Science, Technology, and Engineering Leadership Program: 
Extent of Science Integration and Collaboration in Two Sessions 



October PD Session 

May PD Session 





Somewhat/ 

Not at 




Somewhat/ 

Not at 

Science integration and 


A Lot 

A Little 


All 


A Lot 

A Little 


All 

collaboration 

N 

n 

% 

n 

% 

n 

% 

N 

n 

% 

n 

% 

n 

% 

Elementary: extent of 
science integration into 
school schedule 

13 

3 

23.1 

9 

69.2 

i 

7.7 

9 

4 

44.4 

5 

55.6 

0 

0.0 

Secondary: extent of 
collaboration between 
math and science teachers 

22 

0 

0.0 

12 

54.5 

10 

45.5 

16 

1 

6.3 

6 

37.5 

9 

56.3 

Secondary: extent of 
collaboration between 
math, science and 
technology teachers 

23 

0 

0.0 

10 

43.5 

13 

56.5 

18 

1 

5.6 

11 

61.1 

6 

33.3 


Note. PD = professional development. 
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Table 14 


Science, Technology, and Engineering Leadership Program: 
Change in Extent of Science Integration and Collaboration 



October to May 



Increased 

No Change 

Decreased 

Science integration and collaboration 

N 

n 

% 

n 

% 

n 

% 

Elementary: extent of science integration 
into school schedule 

7 

i 

14.3 

6 

85.7 

0 

0.0 

Secondary: extent of collaboration between 
math and science teachers 

16 

5 

31.3 

9 

56.3 

2 

12.5 

Secondary: extent of collaboration between 
math, science and technology teachers 

16 

5 

31.3 

8 

50.0 

3 

18.8 


Teacher Leaders Reflection 

At the last session in May, the participant survey included open-ended reflective questions about 
the STELP project as a whole. Teacher leaders’ responses are summarized below and shown in 
Table 15. Percentages are calculated from the number of participants who answered the question. 

STELP Impact on Instruction. In an open-ended question that asked, “How has the 
STELP professional development on proficiencies and practices impacted your instruction,” 
more than half (56%) of the 25 participants responded by saying they are including and focusing 
more on the practices. Almost one third (8 of 25, 32%) responded that the STELP project has 
impacted their instruction by having them reflect and improve their own teaching and lessons. 

STELP Impact on Working with Colleagues and Sharing Information. Forty percent 
(10 of 25) responded that sharing the practices, standards, and information acquired is how the 
STELP professional development impacted their work with colleagues at their school, and 9 of 
23 (39%) said they have done so at team and department meetings, followed by 5 of 23 (22%) 
not specifying how they shared, and 4 of 23 (17%) saying they shared informally with 
colleagues. 

Important Gain from STELP. Almost half (11 of 24, 46%) of the respondents stated 
that the most important thing they gained from this STELP project was a better understanding of 
the science practices and learning to integrate the practices into teaching. 

School Science Events. When asked, “how have you used the science and engineering 
practices to upgrade science events at your school,” more than one third (8 of 22, 36%) said that 
they have not had any science event at their school; however 4 of 22 (18%) said they’ve used 
their practices to upgrade their school’s STEM night or STEM conference. 
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Table 15 

Science, Technology, and Engineering Leadership Program: 
Teacher Leaders’ Reflection on the STELP Program 


Impact on instruction (N = 25) 


Reported impact 

Examples 

Participants include 
practices/focus more on 
practices/ (n = 14, 56%) 

• STELP has made me rethink how to re -plan lessons to include the practices. 
Lessons will be more than "covering" a bunch of topics. More depth rather than 
breath. 

• I have found myself trying to incorporate the practices into my everyday 
teaching. 

• I focus on the process skills/practices much more than the content alone. 

Participants reflect on own 
teaching/improve lesson, 
presentation 
(n = 8, 32%) 

• The videos I created for the PowerPoint caused me to be more reflective on 
teaching practice 

• Look more carefully at how I introduce the information to my students 

Impact work with colleagues at your school (N = 25) 

Increased sharing of 
practices/standards/info 
with colleagues 
(n = 10, 40%) 

• It has influenced me to have more conversations with my colleagues about the 
NGSS standards and how they will impact future instruction. 

• I try to share what I've learned with colleagues who will listen 

• I have talked about the new practices with my colleagues at our department 
meetings - to introduce and facilitate the transition to the NGSS 

Ways that practices were shared with colleagues ( N = 23) 

At Team/Department 
meetings ( n = 9, 39%) 

• At team meetings 

• Dining Department meeting 

Shared/Discussed - 
unspecified 
(n = 5, 22%) 

• Have discussed what I have learned/discovered while participating in STELP 
training 

• Discussion 

Informally 
(n = 4, 17%) 

• Casual conversations with other staff members 

• Via e-mail 

. 

Most important thing gained from STELP project ( N = 24) 

Better understanding of 

practices 

(n= 11,46%) 

• A better understanding of the practices and how to integrate them into my 
teaching 

• Better understanding of all the practices. How the practices, standards and 
strands all relate 

• Increased awareness of the NGSS 

Integrating practices into 

teaching 

(n= 11,46%) 

• It has forced me to think about what I am doing in class and how well it is 
aligned with the NGSS 

• Better understanding of how to implement it in my teaching because I have had 
to film it in my class 

• New perspective on what the students should be able to do after leaving my 
classroom. It's not all about content and the textbook AT ALL! 

Networking with 
colleagues (n = 5, 21%) 

• Really enjoyed working with colleagues in county and getting their 
perspectives. 

• Collaboration with other teachers as we transition to NGSS. Our discussions 
really inspired me to make changes 

How have you used the science and engineering practices to upgrade science events at your school? (N = 22) 

Have not had any 
(n = 8, 36%) 

• Have not had any events 

At STEM night/STEM 
conference (n = 4, 18%) 

• Inviting more engineering and technology presenters at STEM night 

• STEM conference at our school-yearly event 
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Question 4 

When and how were the online products made available to MCPS teachers? 

Development and Dissemination of Online Professional Development Products 

In year two, eight teams of teacher leaders created eight online professional development 
products (also called modules) to help other MCPS educators learn about the NGSS practices. 
The online products were introduced to resource teachers in September 2012. 

In year three, the online products were updated, refined, and relaunched in July 2013 to 
approximately 90 secondary science RTs at an RT professional development meeting. The 
meeting consisted of breakout sessions where selected STELP teacher leaders (creators of the 
products) demonstrated a few of the online products to groups through a 55 minute presentation. 
The RTs were encouraged to introduce the online professional development products to their 
school staff in the fall. The online products also were discussed in the fall during an October 
2013 RT meeting. The eight online product modules also were placed on a science SharePoint 
site accessible to all MCPS teachers. 

Development of Online Professional Development Product Training Plans 

Along with the online products, each leadership team produced a detailed training plan to 
coincide with their product. These training plans outlined a way for resource teachers or others 
in a training role to incorporate the module as a tool when training their teachers on a National 
Generation Science practice. A training plan template can be found in Appendix D. 

A summary of the content for each of the training plans can be seen in Appendix E. Included in 
many of the training plans were: viewing all or parts of the product module; a hands-on activity; 
3-2-1 summarizer; sharing and discussions; reviewing student work; capture sheets and 
reflections. 

Question 5 

What was the level of usage of the online professional development products by classroom 
teachers? 

There was no evidence of use, based on the survey data, of the online products among classroom 
teachers; however, we cannot fully and accurately answer this question because the software 
used for the online products was not capable of tracking visitors. A link to an online feedback 
survey was attached to the products’ main page so that participants could leave feedback for each 
product they viewed. These surveys were available throughout year three, including after the 
products were revised at the end of year three. Unfortunately, the response rate for the online 
feedback surveys was zero. This may or may not indicate no usage or low usage of the products, 
since the survey was a separate link and completion was optional. 

Use of Online Professional Development Products Among Secondary Science Resource 
Teachers 

The RTs were shown the products in July 2013 and were encouraged to introduce them to their 
school staff in the fall. RTs were exposed to the products again at a meeting in October. 
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A paper and pencil survey was administered in November 2013 to secondary science RTs at a 
professional development meeting held in six locations. They were asked whether they had used 
the online professional development products and if they had plans to use or share them in the 
future for their own school staff development. Out of 88 RTs, a total of 62 were present at the 
meetings and 54 surveys were completed (this does not count two participants’ incomplete 
surveys), which is an 87% response rate of those attending and 61% of all secondary science 
RTs. A copy of the survey may be found in Appendix C. 

As shown below in Table 16, the 54 RTs responding to the survey were evenly divided between 
middle school and high school levels and most were science RTs (70%). Almost half (48%) 
attended the summer and October RT meetings where the online products were demonstrated or 
discussed, and more than one third (39%) attended only the summer meeting. 

Table 16 


Science, Technology, and Engineering Leadership Program: 
Resource Teacher Survey Participants 




All 


(N 

= 54) 

Characteristics 

n 

% 

RT position 

Middle school 

26 

48.1 

High school 

26 

48.1 

Both 

1 

1.9 

No answer 

2 

3.7 

RT subject 

Science 

38 

70.4 

Technology + Engineering 3 

10 

18.5 

Both 

5 

9.3 

No answer 

2 

3.7 

Prior 2013 meeting attendance 

Summer RT meeting only 

21 

38.9 

October RT meeting only 

1 

1.9 

Both Summer and October 

26 

48.1 


“Eight of the ten are at the high school level. 


About one half (52%) of the RTs reported that they had individually viewed one or more of the 
online professional development products outside of RT meetings (Table 17). Slightly more 
middle school RTs reported this compared to high school RTs (17 of 26, 65% versus 11 of 27, 
41%). 

Over one third of the RTs (39%) reported that they have shared one or more of the online 
professional development products with other staff at their school. One half of the middle school 
RTs reported sharing (13 of 26, 50%), and nearly one third of high school RTs reported sharing 
(8 of 27, 30%) with other staff at their school (Table 17). 
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Table 17 


Science, Technology, and Engineering Leadership Program: 
Viewed the Online Modules Outside of Resource Teacher Trainings 


Method of viewing modules 
(i.e., products) 

All RTs 

(N = 54) a 

Middle School 
RTs 

(N = 26) 

High School 
RTs 

(N = 27) b 

n 

% 

n 

% 

n 

% 

Viewed PD modules 

Yes 

28 

51.9 

17 

65.4 

ii 

40.7 

individually 

No 

26 

48.1 

9 

34.6 

16 

59.3 

Shared PD modules 

Yes 

21 

38.9 

13 

50.0 

8 

29.6 


No 

33 

61.1 

13 

50.0 

19 

70.4 


Note. Modules = products; PD = professional development. 

d One respondent did not answer level of school; they were included with all RTs. 

b The RT located at both levels was combined with high school RTs. 


A closer look reveals that the most commonly viewed and shared product among responding 
resource teachers was the product for Practice 4: Analyzing and Interpreting Data with one 
fourth of RTs individually viewing it (14 of 54, 26%) and almost one fifth sharing it (10 of 54, 
19%). This was followed by Practice 7: Engaging in Argument from Evidence (11 of 54, 20% 
viewing, and 10 of 54, 19% sharing) and Practice 1: Asking Questions and Defining Problems 
(10 of 54, 19% viewing, and 9 of 54, 17% sharing). Upon further examination, three of the RTs 
indicated that they did not view the products individually, but did share them (Table 18). 

Table 18 


Science, Technology, and Engineering Leadership Program: 

Online Modules Viewed and Shared Outside of Resource Teacher Trainings 



Modules Viewed 
Individually 

Modules Shared 

Modules (i.e., products) 

All RTs 
(N = 54 ) 

All RTs 
(N = 54 ) 


n 

% 

n 

% 

PI: Asking Questions and Defining Problems. 

10 

18.5 

9 

16.7 

P2: Developing and Using Models. 

6 

11.1 

5 

9.3 

P3: Planning and Carrying Out Investigations. 

7 

13.0 

4 

7.4 

P4: Analyzing and Interpreting Data. 

14 

25.9 

10 

18.5 

P5: Using Mathematics and Computational Thinking. 

4 

7.4 

4 

7.4 

P6: Constructing Explanations and Designing 
Solutions. 

2 

3.7 

3 

5.6 

P7: Engaging in Argument from Evidence. 

11 

20.4 

10 

18.5 

P8: Obtaining, Evaluating and Communicating 
Information. 

3 

5.6 

5 

9.3 

I have, but don’t remember which one(s). 

3 

5.6 

3 

5.6 

No, I have not completed/shared any. 

26 

48.1 

34 

63.0 


Of the 54 participants, 21 (39%) of them indicated that they shared at least one online product 
with staff (Table 19). Those who shared were then asked to indicate the way(s) in which they 
shared the online products. More than half (13 of 21, 62%) indicated that they made staff aware 
of the products in a staff meeting or training, and 9 of the 21 (43%) indicated that they went 
through a product as a group. 
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Table 19 


Science, Technology, and Engineering Leadership Program: 
Ways Online Modules Were Shared by Resource Teachers 


Methods of sharing module (i.e., product) information 

All RTs 

(A = 2 1 ) 

n 

% 

Made staff aware of the modules in a staff meeting/training 

13 

61.9 

Went through a module(s) as a group during a meeting/training 

9 

42.9 

E-mailed link to staff 

4 

19.0 

Used the module’s training plan 

4 

19.0 

Other 

1 

4.8 


Note. N = 21 respondents shared at least one online module with staff. 


A third (33%) of the participants said that they definitely will complete a module (i.e., an online 
professional development product) in the future, with another third (35%) saying they probably 
will (Table 20). Several reported that they probably will not, but no one indicated that they 
definitely will not. More than a third (42%) said they definitely will share a product module with 
their staff in the future, and more than a quarter (28%) said they probably will. 

Half reported that they definitely will or probably will use the product module’s training plan in 
the future, and another third said they might or might not (35%). 

Table 20 


Science, Technology, and Engineering Leadership Program: 
Usefulness of Online Modules 



Resource Teachers 


Definitely 

Probably 

Might 

or 

Probably 

Definitely 



Will 

Will 

Might Not 

Will Not 

Will Not 

How likely will you be to. . . 

n 

% 

n 

% 

n 

% 

n 

% 

n 

% 

Complete any of the modules after today? 
(A =52) 

17 

32.7 

18 

34.6 

9 

17.3 

8 

15.4 

0 

0.0 

Share any additional modules with staff at 

21 

42.0 

14 

28.0 

11 

22.0 

A 

8.0 

0 

0.0 

your school? (A = 50) 


Use the module’s training plan? 
(A = 48) 

8 

16.7 

16 

33.3 

17 

35.4 

7 

14.6 

0 

0.0 


Finally, RTs were asked to describe how they might take back and use the online professional 
products. Of the 30 participants who gave a response, more than half of them (17 of 30, 57%) 
replied that they would share at a department meeting, department training, professional 
development, or professional learning community. Four of the participants replied that they 
would incorporate them or use them to support student learning objectives (SLO’s), and three 
stated that they don’t have the time to take them back and use. 
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Question 6 

What was the impact of the online professional development products on teachers who accessed 
them? 

Perception of the Online Professional Development Products 

When the online products were published at the end of year two, a link to an online feedback 
survey was attached to the products’ website main page so that participants could provide 
feedback on the usage, ease of navigation, and helpfulness for each product they viewed (see 
Appendix B for an example). The surveys continued to be available throughout year three, 
including after the products were revised at the end of year three. The response rate for the 
feedback survey was zero. 

Usefulness of Online Professional Development Products among Secondary Science Resource 
Teachers 

Secondary science RTs who attended a November 2013 RT meeting were asked how useful they 
thought the online products were. The majority of respondents indicated that they thought the 
products were extremely or very useful for classroom teachers (20 of 34, 59%), and more than 
one third indicated they were somewhat useful (38%) (Table 21). Almost three fourths (27 of 
37, 73%) thought the products were extremely or very useful for their role as RTs. 

Table 21 


Science, Technology, and Engineering Leadership Program: 
Usefulness of Online Product Modules 


How useful do you find the professional 
development modules (i.e., products)... 

Resource Teachers 

Not Very 

Extremely/ Somewhat Useful/Not 

Very Useful Useful Useful at All 

n % n % n % 

For classroom teachers? ( N = 34) 

20 58.8 13 38.2 1 2.9 

For your role as a STE resource teachers? ( N = 37) 

27 73.0 9 24.3 1 2.7 


Recommendations 

• Continue to: a) promote and explore additional ways and formats to ensure the 
widespread and effective use of the online products; and b) explore additional 
ways in which the online products may be used by resource teachers. Include 
examples of how they can share with their staff and effectively use them in their 
staff meetings and professional development. 

• Elicit user feedback on the quality and usefulness of the online products so that 
modifications can be made to address concerns or problems. 

• Encourage user feedback on the ways that educators are using the online products 
so that best practices can be shared. 
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Appendix A: Training Sessions’ Feedback Surveys 

Feedback Survey I 

Science Technology Engineering Leadership Program (STELP) 

Year 3, Oct. 26, 2012 

As part of your involvement in the Science, Technology, and Engineering Leadership Program 
(STELP), we will be asking you to provide information and feedback about the program. Your 
input is very important to the evaluation of the program and it will help guide administrators in 
planning and implementation. 

This is the first in a series of surveys that will be given after each training session during year 
three of the program, thus our need to collect names to keep each participant’s surveys together; 
however, your answers are strictly confidential (i.e. answers will not be linked to individual 
names when reporting). Completed surveys are collected by OSA staff. Summaries and non- 
identifiable answers are reported by OSA staff to the STE program staff and as part of the year 
three final evaluation report. 


Name: 


School: 

Team: 

1. Please cheek your current position(s) (enter all that apply) 
Elementary Level: O 


□ 

Kindergarten Teacher 

1 1 ES Technology Education Teacher 


□ 

Grade 1 Teacher 

1 1 Staff Development Teacher 


□ 

Grade 2 Teacher 

i ] Special Ed Teacher (specify grades 

) 

□ 

Grade 3 Teacher 

1 ~| Focus Teacher (specify grades + subject 

) 

□ 

Grade 4 Teacher 

1 ES Science Teacher (specify grades 

) 

□ 

n 

Grade 5 Teacher 
Other (please specify 

) 


Secondary School Level: O 

n Science Teacher (specify srade(s) or subiect(s) 

) 

□ 

Technology Education Teacher 



I I Staff Development Teacher 

I I Special Ed Teacher (specify grades ) 

I I Other (please specify ) 
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2. How many years have you been in your current classroom teacher position, or SDT 

position, (including the current year)? 

3. Please indicate any Leadership Roles you currently have: 

I | Grade Team Leader 
I I Content Specialist 
I I Secondary Science Resource Teacher 
I I Secondary Technology Education Resource Teacher 

I I Other (please specify ) 

4. Do you have a college degree or teacher certification in science, technology, and/or 
engineering? 

I I Yes O No 

5. Have you participated in the Howard Hughes STELP Teacher Leader Program 
before? 

I | No, this is my 1 st year 

-Did you participate in the September 21 st new participant training? Q Yes I I No 
I | Yes, this will be my 2nd year 
I | Yes, this will be my 3 rd year 

New STELP Participants (answer 6 thru 6b): 

6. Do you have any experience leading or developing professional development? 

I | Yes O No 


6a. If yes, did any of your experiences include leading or developing online professional 
development? 

I I Yes O No 

6b. If yes, please describe the extent of your experience leading or developing trainings. 
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7. If you are an elementary teacher, to what extent is science integrated into the 
students’ schedule at your school? 

i | A lot ^Somewhat OA little QNot at all 


8. If you are a secondary school teacher, to what extent are math and science teachers 
collaborating with each other on lesson planning and looking at student work? 

I | A lot ^Somewhat OA little QNot at all 


9. If you are a secondary school teacher, to what extent are technology education, 
math, and science teachers collaborating with each other on lesson planning and 
looking at student work? 

I I A lot ^Somewhat OA little QNot at all 


(turn page to continue) 
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Feedback on Today’s Training 


10. Please indicate how much you agree or disagree with the following items by checking 
the appropriate box. 



Strongly 

Agree 

Agree 

Disagree 

Strongly 

Disagree 

a. The goals of today’s training were clear. 

□ 

□ 

□ 

□ 

b. The objectives of today’s training were met. 

□ 

□ 

□ 

□ 

c. The trainers were knowledgeable and well- 
prepared. 

□ 

□ 

□ 

□ 

d. An environment was created in which I felt 
comfortable taking risks (i.e., asking 
questions, expressing my ideas, working with 
unfamiliar content). 

□ 

□ 

□ 

□ 

e. Opportunities were provided for me to 
process and reflect upon the application of 
the knowledge and skills learned. 

□ 

□ 

□ 

□ 

f. My questions during the training today were 
answered adequately. 

□ 

□ 

□ 

□ 

g. As a result of today’s training, I have gained 
information and skills that will help me in 
this role as an STE leader. 

□ 

□ 

□ 

□ 

h. Critiquing existing PD modules was helpful 
in understanding the expectations for creating 
online PD 

□ 

□ 

□ 

□ 

i. Working in practice groups was helpful in 
building my knowledge of a specific practice 
and preparing me to develop a training for 
other MCPS teachers related to the practice 

□ 

□ 

□ 

□ 

j. The training session today helped me build 
my understanding of the proficiencies of 
Science and Engineering as described in the 
NGSS. (If you already have a thorough 
understanding of the proficiencies, please 
write “NA”.) 

□ 

□ 

□ 

□ 

k. The training session today helped me build 
my understanding of the proficiencies at 
different grade levels from the ones I teach. 

□ 

□ 

□ 

□ 
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11. Please elaborate on any of the statements above. (Indicate the corresponding letter 
item(s) with your comments or explanation/elaboration of your rating. 


12. Please indicate your level of skill or understanding with the following items by 
checking the appropriate box. 



Basic/ 

Beginner 

Proficient 

Advanced 

a. Using video style camera technology for 
capturing student learning. 

□ 

□ 

□ 

b. Using Windows Movie Maker software for 

□ 

□ 

□ 

producing videos for online PD modules 

c. Using Microsoft Office Power Point 




capabilities for creating user-friendly, user- 
choice navigation for online professional 
development products. 

□ 

□ 

□ 

d. Understanding and articulating the 

proficiencies and their relationship to the 
MCPS STEM vision 

□ 

□ 

□ 

e. Understanding and articulating the 

proficiencies of Science and Engineering as 
described by the Next Generation Science 

□ 

□ 

□ 


Standards 


13. What is the most important thing you gained from this training? 


14. Was there anything about the training you think would have been more effective if it 
were done differently? 
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Overall Feedback on the STELP Project 


15. Please indicate how much you agree or disagree with the following items by 
checking the appropriate box. 



Strongly 

Agree 

Agree 

Disagree 

Strongly 

Disagree 

a. The expectations for the 

professional development products 
are clear. 

□ 

□ 

□ 

□ 

b. The expectations for next steps are 
clear. 

□ 

□ 

□ 

□ 


16. Please elaborate on any of the statements above. (Indicate the corresponding letter 
item(s) with your comments or explanation/elaboration of your rating. 


17. Are there any additional resources or supports that you think you will need to fulfill 
the teacher leader role as you develop your professional development product? 


Thank you for your help. 
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Feedback Survey II 

Science Technology Engineering Leadership Program (STELP) 

Year 3, Dec. 14, 2012 

As part of your involvement in the Science, Technology, and Engineering Leadership Program 
(STELP), we will be asking you to provide information and feedback about the program. Your 
input is very important to the evaluation of the program and it will help guide administrators in 
planning and implementation. 

This is the second in a series of surveys that will be given after each training session during year 
three of the program, thus our need to collect names to keep each participant’s surveys together; 
however, your answers are strictly confidential (i.e. answers will not be linked to individual 
names when reporting). Completed surveys are collected by OSA staff. Summaries and non- 
identifiable answers are reported by OSA staff to the STE program staff and as part of the year 
three final evaluation report. 


Name: School: 

Team (Practice Assigned): 


Feedback on Today’s Training 


1. Please indicate how much you agree or disagree with the following items by checking the 
appropriate box. 



Strongly 

Agree 

Agree 

Disagree 

Strongly 

Disagree 

a. The goals of today’s training were clear. 

□ 

□ 

□ 

□ 

b. The objectives of today’s training were met. 

□ 

□ 

□ 

□ 

c. The trainers were knowledgeable and well- 
prepared. 

□ 

□ 

□ 

□ 

d. An environment was created in which I felt 





comfortable taking risks (i.e., asking questions, 
expressing my ideas, working with unfamiliar 
content). 

□ 

□ 

□ 

□ 

e. Opportunities were provided for me to process and 
reflect upon the application of the knowledge and 
skills learned. 

□ 

□ 

□ 

□ 

f. My questions during the training today were 
answered adequately. 

□ 

□ 

□ 

□ 

g. As a result of today’s training, I have gained 

information and skills that will help me in this role 

□ 

□ 

□ 

□ 


as an STE leader. 
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2. Please indicate how helpful you found the following items by checking the 
appropriate box. 




Very 

Somewhat 

Not Very 

Not at all 



Helpful 

Helpful 

Helpful 

Helpful 

a. 

How helpful was the capture sheet in 
developing your online PD? 

□ 

□ 

□ 

□ 

b. 

How helpful was viewing the proposed 
plan of delivery? 

□ 

□ 

□ 

□ 

c. 

How helpful was the development of a 
training plan for use of delivering your 

□ 

□ 

□ 

□ 


product to school level team meetings? 


3. What is the most important thing you gained from this training? 


4. Was there anything about the training you think would have been more effective if it 
were done differently? 
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Overall Feedback on the STELP Project 


5. Please indicate how much you agree or disagree with the following items by 
checking the appropriate box. 



Strongly 

Agree 

Agree 

Disagree 

Strongly 

Disagree 

a. The expectations for the 

professional development products 
are clear. 

□ 

□ 

□ 

□ 

b. The expectations for the delivery 
of the professional development 
products are clear 

□ 

□ 

□ 

□ 

c. The expectations for what our 
teacher team is to accomplish for 
this project is clear. 

□ 

□ 

□ 

□ 

d. The expectations for next steps are 
clear. 

□ 

□ 

□ 

□ 

e. I feel good about the direction my 
team’s heading in 

□ 

□ 

□ 

□ 

f. I believe today’s session has 





helped with our teacher team’s 
progress as we develop our 

□ 

□ 

□ 

□ 


6. Are there any additional resources or supports that you think you will need to fulfill the 
teacher leader role as you develop your professional development product? 


7. Do you have any other comments about the training or product development? 


Thank you for your help. 
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Feedback Survey III 

Science Technology Engineering Leadership Program (STELP) 

Year 3, Feb 8, 2013 

As part of your involvement in the Science, Technology, and Engineering Leadership Program 
(STELP), we will be asking you to provide information and feedback about the program. Your 
input is very important to the evaluation of the program and it will help guide administrators in 
planning and implementation. 

This is the third in a series of surveys that will be given after each training session during year 
three of the program, thus our need to collect names to keep each participant’s surveys together; 
however, your answers are strictly confidential (i.e. answers will not be linked to individual 
names when reporting). Completed surveys are collected by OSA staff. Summaries and non- 
identifiable answers are reported by OSA staff to the STE program staff and as part of the year 
three final evaluation report. 


Name: School: 

Team (Practice Assigned): 


Feedback on Today’s Training 


1. Please indicate how helpful you found the following items by checking the 
appropriate box. 




Very 

Somewhat 

Not Very 

Not at all 



Helpful 

Helpful 

Helpful 

Helpful 

a. 

How helpful was the carousel to determine 
additions needed to your product module? 

□ 

□ 

□ 

□ 

b. 

How helpful was the development of a 
training plan for use of delivering your 
product to school level team meetings? 

□ 

□ 

□ 

□ 

c. 

How helpful was it to determine a timeline 
of action steps to be taken between now 
and April? 

□ 

□ 

□ 

□ 

d. 

How helpful was it to work with your 

□ 

□ 

□ 

□ 


assigned instructional specialist? 
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2. What is the most important thing you gained from this training? 



3. Was there anything about the training you think would have been more effective if it 
were done differently? 



Overall Feedback on the STELP Project 


4. Please indicate how much you agree or disagree with the following items by 
checking the appropriate box. 



Strongly 

Agree 

Agree 

Disagree 

Strongly 

Disagree 

a. The expectations for the 

professional development products 
are clear. 

□ 

□ 

□ 

□ 

b. The proposed deadlines and due 
dates are realistic 

□ 

□ 

□ 

□ 

c. The expectations for the delivery 
of the professional development 
products are clear 

□ 

□ 

□ 

□ 

d. The expectations for what our 
teacher team is to accomplish for 
this project is clear. 

□ 

□ 

□ 

□ 

e. The expectations for next steps are 
clear. 

□ 

□ 

□ 

□ 

f. I feel good about our team’s 
product so far 

□ 

□ 

□ 

□ 

g. I feel good about the progress my 
teacher team has made so far. 

□ 

□ 

□ 

□ 

h. I believe today’s session has 





helped with our teacher team’s 
progress as we develop our 

□ 

□ 

□ 

□ 
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5. How would you describe your team’s progress on your professional development 
product? 

□ Not as far along as we would like; running behind 

□ Right on schedule 

□ Ahead of our planned timeline 

6. Are there any additional resources or supports that you think you will need to fulfill the 
teacher leader role as you develop your professional development product? 


7. Do you have any other comments about the training or product development? 


Thank you for your help. 
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Feedback Survey IV 

Science Technology Engineering Leadership Program (STELP) 

Year 3, May 10, 2013 

This is the fourth in a series of surveys that you have been asked to complete as part of your 
involvement in the Science, Technology, and Engineering Leadership Program (STELP) this 
year. Your continued input is very important to the evaluation of the program and it also will 
continue to guide administrators in planning and implementation. 

Since surveys have been administered after several training sessions during year three of the 
program, we have needed to collect names in order to keep each participant’s surveys together; 
however, your answers are strictly confidential (i.e. answers will not be linked to individual 
names when reporting). Completed surveys are collected by OSA staff. Summaries and non- 
identifiable answers are reported by OSA staff to the STE program staff and as part of the year 
three final evaluation report. 


Name: School: 

Team (Practice Assigned): 


Feedback on Today’s Training 


1. Please indicate how helpful you found the following items by checking the 
appropriate box. 




Very 

Somewhat 

Not Very 

Not at all 



Helpful 

Helpful 

Helpful 

Helpful 

a. 

How helpful was the development of a 
training plan for future delivery of the 
product to school level team meetings? 

□ 

□ 

□ 

□ 

b. 

How helpful was the STELP checklist in 
finalizing your product? 

□ 

□ 

□ 

□ 

c. 

How helpful was it to work with your 
assigned instructional specialist? 

□ 

□ 

□ 

□ 
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Overall Feedback on the STELP Project 


2. Please indicate how much you agree or disagree with the following items by 
checking the appropriate box. 



Strongly 

Agree 

Agree 

Disagree 

Strongly 

Disagree 

a. I believe today’s session has 





helped with our teacher team’s 
progress as we finalized our 

□ 

□ 

□ 

□ 

professional development product, 
b. The expectations for the 

professional development products 
are clear. 

□ 

□ 

□ 

□ 

c. The proposed due dates were 
realistic. 

□ 

□ 

□ 

□ 

d. The expectations for the delivery 
of the professional development 
products were clear. 

□ 

□ 

□ 

□ 

e. The expectations for what our 
teacher team was to accomplish for 
this project was clear. 

□ 

□ 

□ 

□ 

f. The expectations for next steps are 
clear. 

□ 

□ 

□ 

□ 

g. I feel good about our team’s 
product so far. 

□ 

□ 

□ 

□ 

h. I feel good about the progress my 
teacher team has made this year. 

□ 

□ 

□ 

□ 


3. How would you describe your team’s progress on your professional development 
product? 

□ Not as far along as we would like; running behind 

□ Right on schedule 

□ Ahead of our planned timeline 
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Proficiencies 


4. Please indicate your level of skill or understanding with the following items by 
checking the appropriate box. 



Basic/ 

Beginner 

Proficient 

Advanced 

a. Using video style camera technology for 
capturing student learning. 

□ 

□ 

□ 

b. Using Windows Movie Maker software for 

□ 

□ 

□ 

producing videos for online PD modules 

c. Using Microsoft Office Power Point 




capabilities for creating user-friendly, user- 
choice navigation for online professional 
development products. 

□ 

□ 

□ 

d. Understanding and articulating the proficiencies 
and their relationship to the MCPS STEM 

□ 

□ 

□ 

vision 




e. Understanding and articulating the proficiencies 
of Science and Engineering as described by the 

□ 

□ 

□ 


Next Generation Science Standards 

Integration 

5. If you are an elementary teacher, to what extent is science integrated into the students’ 
schedule at your school? 

I | A lot ^Somewhat OA little QNot at all 

6. If you are a secondary school teacher, to what extent are math and science teachers 
collaborating with each other on lesson planning and looking at student work? 

I I A lot ^Somewhat OA little QNot at all 


7. If you are a secondary school teacher, to what extent are technology education, math, 
and science teachers collaborating with each other on lesson planning and looking at 
student work? 

I I A lot ^Somewhat OA little QNot at all 


Program Evaluation Unit 


41 


Evaluation of STELP, Year Three 






Montgomery County Public Schools 


Office of Shared Accountability 


Reflection 


8. How has the STELP professional development on science and engineering proficiencies 
{Ready, Set, SCIENCE !) and scientific and engineering practices impacted your 
instruction? 



9. How has the STELP professional development on science and engineering proficiencies 
{Ready, Set, SCIENCE !) and scientific and engineering practices impacted your work 
with colleagues at your school? 



10. How have you shared what you have learned about the science and engineering 
practices with colleagues? 
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11. How have you used the science and engineering practices to upgrade science events at 
your school? 



12. What is the most important thing you have gained from this STELP project? 



13. Do you have any other comments about the training, product development or STELP 
project? 



Thank you for your help! 
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Appendix B: Example of Online Module Survey Questions 

Feedback Survey: Analyzing and Interpreting Data - 
Online Professional Development Module 

After you have completed all the portions of the Analyzing and Interpreting Data module that 
you intend to complete, please complete this anonymous survey. The Office of Shared 
Accountability (OSA) is conducting this survey to provide feedback for the Howard Hughes 
STELP Grant. Complete the survey only ONE time, even if you have exited and come back to 
view in other sittings. If possible, please fill out this feedback survey individually. 

**** An SW er the survey questions only about the module “Analyzing and Interpreting 
Data.” 

Did you view this Online Professional Development module 

□ Individually 

□ With a Group 

□ Both 

If you are taking the survey as a group, please indicate the type of group 

***SKIP if you are taking this survey individually 

□ Grade Level Team 

□ Math Team 

□ Technology Team 

□ Other 

What is the level of the school where you work? 

□ Elementary 

□ Middle 

□ High 

□ Two levels or all three levels 

What is your current position? 

(check all that apply) 

□ Kindergarten Teacher 

□ 1 st Grade Teacher 

□ 2 nd Grade Teacher 

□ 3 rd Grade Teacher 

□ 4 th Grade Teacher 

□ 5 th Grade Teacher 

□ Secondary Science Teacher 

□ Secondary Math Teacher 

□ Secondary Technology Teacher 

□ Science Resource Teacher 

□ Staff Development Teacher 

□ Special Education Teacher 

□ Other: 
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How did you first hear about these Online Professional Development Modules? 

(Check all that apply) 

□ Resource Teacher 

□ Staff Development Teacher 

□ Colleague from same school 

□ Colleague from different school 

□ E-mail from Science Department 

□ Newsletter from Science Department 

□ EIC link 

□ I am a STELP Participant 

□ Colleague who is a STELP Participant 

□ Other 

Have you viewed 

(answer only for the Analyzing and Interpreting Data module) 

□ The entire module (i.e. all portions provided) 

□ Most of the module (i.e. skipping 1 or 2 portions) 

□ Some of the module (i.e. skipping 3 or more portions/only viewing 1-2 portions) 

How many sittings did you use to view THIS online module? 

□ Viewed it all in 1 sitting 

□ Viewed it in 2 sittings 

□ Viewed it in 3-4 sittings 

□ Viewed it in 5 or more sittings 

If you skipped any portions of this online module, please state which parts and why you 
decided to skip them 
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Indicate how much you agree or disagree with the following statements about THIS online 
module: Analyzing and Interpreting Data. 



Strongly 

Agree 

Agree 

Disagree 

Strongly 

Disagree 

I had no problems accessing the online training 
module. 

□ 

□ 

□ 

□ 

The online training module was easy to navigate. 

□ 

□ 

□ 

□ 

The online module was user friendly. 

□ 

□ 

□ 

□ 

The organization of this online module was good. 

□ 

□ 

□ 

□ 

The production quality of the online module was 
professional. 

□ 

□ 

□ 

□ 

The explanation(s) of the Analyzing and Interpreting 
Data science practice was easy to understand. 

□ 

□ 

□ 

□ 

The Checks for Understanding/Post-Assessment(s) 
were useful. 

□ 

□ 

□ 

□ 

I learned new information by viewing this module. 

□ 

□ 

□ 

□ 

I have a better understanding of this science practice 
after viewing this module. 

□ 

□ 

□ 

□ 

I have learned new ways to support students in their 
development of this practice. 

□ 

□ 

□ 

□ 

I will use what I have learned in this module in my 
classroom. 

□ 

□ 

□ 

□ 


Would you recommend this Analyzing and Interpreting Data training module to other 
teachers? 

□ Definitely would 

□ Probably would 

□ Might or Might Not 

□ Probably would not 

□ Definitely would not 

Have you viewed any of the other eight online modules? 

□ Yes, all of them 

□ Yes, some of them 

□ No, but I plan to 

□ No 

Please provide any comments on what you like/found helpful about this online module 
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Please describe any dissatisfaction you had or any problems you encountered with this 
online module. 


How likely will you incorporate any of the strategies that you learned from this online 
professional development module into your classroom? 

□ Definitely will 

□ Probably will 

□ Might or might not 

□ Probably will not 

□ Definitely will not 

Please describe how you might incorporate this practice into your classroom. 


How likely is it that you will share any of the information that you learned from this online 
professional development module? 

□ Very likely 

□ Somewhat likely 

□ Might or might not 

□ Somewhat unlikely 

□ Very unlikely 


Please describe your plans for sharing the information (e.g. with whom will you share it, in 
what format?) 
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Appendix C: Secondary Science Resource Teacher Survey 

Survey of Online Professional Development Module Usage 
Secondary Science Resource Teachers 
November 6, 2013 

The Office of Shared Accountability (OSA) is conducting this survey as part of a grant funded 
by Howard Hughes. OSA is responsible for the evaluation component of the grant; this survey is 
an integral part of the data collection. This survey will be anonymous; there is no need for your 
name. 

1. Which of the following did you attend (check all that apply) 

I | The Summer 2013 STE RT training 
I | The October 2013 STE RT training 
I | Neither of these 

2. Have you viewed any of the Science and Engineering Practice Online Professional 
Development Modules? (Do not count the RT trainings) 

n Yes EH No EH Not sure 

3. Please indicate which of the following Professional Development Modules, if any, you 
have completed as an individual (Do not count the RT trainings. Check all that apply). 

EH Asking Questions and Defining Problems (practice 1) 

EH Developing and Using Models (practice 2) 

EH Planning and Carrying Out Investigations (practice 3) 

EH Analyzing and Interpreting Data (practice 4) 

EH Using Mathematics and Computational Thinking (practice 5) 

EH Constructing Explanations and Designing Solutions (practice 6) 

EH Engaging in Argument from Evidence (practice 7) 

EH Obtaining, Evaluating and Communicating Information (practice 8) 

EH I have, but don’t remember which one(s) 

EH No, I have not completed any 
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4. Have you shared any of the Online Professional Development Modules with other staff at 
your school? Please indicate which ones, if any, you have shared (check all that apply) 

I | Asking Questions and Defining Problems (practice 1) 

I I Developing and Using Models (practice 2) 

I I Planning and Carrying Out Investigations (practice 3) 

I | Analyzing and Interpreting Data (practice 4) 

I I Using Mathematics and Computational Thinking (practice 5) 

I I Constructing Explanations and Designing Solutions (practice 6) 

I | Engaging in Argument from Evidence (practice 7) 

I I Obtaining, Evaluating and Communicating Information (practice 8) 

I I I have, but don’t remember which one(s) 

I I No, I have not shared any 

5. If you have shared, please indicate the way(s) in which you shared the Online 
professional development modules, (check all that apply) 

I | E-mailed the location of the online pd modules to staff 
I | Made staff aware of the modules in a staff meeting/training 
I I Went through a module(s) as a group during a meeting/training 
I I Used the module’s training plan 

I I Other (please describe) 


6. How useful do you find the professional development modules 



Extremely 

Useful 

Very 

Useful 

Somewhat 

Useful 

Not Very 
Useful 

Not Useful 
at All 

For classroom teachers 

□ 

□ 

□ 

□ 

□ 

For your role as a STE 
resource teacher 

□ 

□ 

□ 

□ 

□ 
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7. How likely will you be to 



Definitely 

will 

Probably 

Will 

Might or 
Might Not 

Probably Will 
Not 

Definitely 
Will Not 

Complete any of the 
modules after today? 

□ 

□ 

□ 

□ 

□ 

Share any or additional 
modules with staff at 
your school? 

□ 

□ 

□ 

□ 

□ 

Use the module’s 
training plan? 

□ 

□ 

□ 

□ 

□ 


8. Please describe how you might take back and use the online professional modules in the 
future. 


9. Are you a Resource Teacher in a Q Middle School Q High School? 


10. Are you a Q Science or O Technology & Engineering Resource Teacher? 

Thank you for your valuable feedback! 
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Appendix D: Module Training Plan Template 


STELP Online PD Training Plan 


MCPS STEM Vision 

All students achieve full science, technology, engineering, and mathematics (STEM) literacy through seamlessly integrated 
instruction that is project/problem and standards-based. STEM literate students are critical thinkers who are able to solve non- 
routine problems in a globally competitive society. 

Session Title: Practice 


Outcome(s) 

By the end of today's professional development, participants will.... 


Mastery of outcome(s) will be measured by.... 


Time 

(Minutes) 

Content 

Including talking points 

Format Plan 

Who 

Resources 

Needed 
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Appendix E: Summary of Content of STELP Online Professional Development Training Plans 


Science & Engineering 
Framework Practice # 

Outcomes: 

Mastery of outcomes measured by: 

Practice 1: Asking 
Questions and Defining 
Problems 

• Identify the importance of questioning in instruction 

• Identify ways to implement strategies to refine 
questioning 

• Identify the relationship between well-defined 
questioning and effective and engaging instruction 

Reviewing section 2 of the 3-2-1 summarizer (write two 
ways you could incorporate asking questions and defining 
problems into your instruction). Participants will modify an 
existing lesson to better incorporate asking questions and 
defining problems 

Practice 1 Training Plan includes: mystery box activity; view product module; think-pair-share; explain; work on lesson plan; 3-2-1 summarizer; follow-up 
assignment 

Practice 2: Developing 
and Using Models 

Develop and use models with students to 

• deepen understanding of concept 

• communicate with others 

• make predictions and ask questions 

• redesign to solve problem 

Participants will complete a reflection and discuss with 
colleagues how models are already used in their 
classrooms and how incorporating models will deepen 
understanding. 

Practice 2 Training Plan includes: view product module; discuss; reflect; choice of think-pair-share, graffiti wall, exit card or sentence completion 

Practice 3: Planning and 
Carrying Out 
Investigations 

Identify ways that student-centered planning provides 
opportunities for students to plan and carry out investigations as 
reflected/described in the NGSS 

Participants will record examples of how students plan and 
carry out investigations in classrooms as described in the 
NGG 

Practice 3 Training Plan includes: round robin pre -assessment, side-by-side comparison; view product module examples; KWL (what know, what want to 
learn, what have learned) 

Practice 4: Analyzing and 
Interpreting Data 

• Identify methods of data analysis and interpretation 
utilized in STE classrooms 

• Identify methods of data analysis and interpretation that 
student currently practice effectively, and methods that 
teachers would like to improve or expand within their 
classrooms 

• Describe examples of data analysis and interpretation 
related to specific content area and ways that the practice 
might be improved/expanded. 

Participants will work in groups to discuss methods of 
Analyzing and Interpreting Data that are currently used in 
STE classrooms. 

Participants will discuss and share suggestions for 
expanding/improving implementation of the practice. 

Practice 4 Training Plan includes: card sort activity pre-assessment; view product module; study and work on sample lessons; 3-2-1 summarizer 
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Science & Engineering 
Framework Practice # 

Outcomes: 

Mastery of outcomes measured by: 

Practice 5: Using 
Mathematics and 
Computational Thinking 

• Obtain knowledge of using mathematics and 
computational thinking 

• Site an example of student engagement in the practice 

• Adapt a current instructional sequence with the 
knowledge of the practice 

n/a 

Practice 5 Training Plan includes: sharing student work or lesson example; text marking activity while reading about practice; view module and answer 
questions, reflect, share , 3-2-lsummarizer 

Practice 6: Constructing 
Explanations and 
Designing Solutions 

Explain the importance of including both aspects of constructing 
explanations and designing solutions in all Science, Technology 
and Engineering classrooms. 

Identify strategies for incorporating both constructing 
explanations and designing solutions into Science, 
Technology and Engineering classrooms 

Practice 6 Training Plan includes: pre-assessment within module, review practices, share/discuss/reflect, view module, review and discuss parts of module, 
post assessment within module 

Practice?: Engaging in 
Argument from Evidence 

• Obtain knowledge of Practice 7 Engaging in Argument 
from Evidence 

• Site an example of student engagement in the practice 

• Adapt a current instructional sequence with the 
knowledge of the practice 

n/a 

Practice 7 Training Plan includes: completing anticipation guide, sharing an example, view training module, share/discuss, complete capture sheet, reflect 

Practice 8: Obtaining, 
Evaluating, 
Communicating 
Information 

Not specified 

Not specified 

Practice 8 Training Plan include resources: 3-2-1 Summarizer, Capture Sheet, Look-fors 
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